Trusting Remote Users... Can They Identify Problems Without Involving Usability Experts?

Based on our belief that critical incident data, observed during usage and associated closely with specific task performance are the most useful kind of formative evaluation data for finding and fixing usability problems, we developed a Remote Usability Evaluation Method (RUEM) that involves real users self-reporting critical incidents encountered in real tasks performed in their normal working environments without the intervention of evaluators. In our exploratory study we observed that users were able to identify, report, and rate the severity level of their own critical incidents with only brief training. Our Study in a Nutshell There are several very different approaches to remote usability evaluation—getting field usability data at home—each with its own advantages, drawbacks, benefits, and costs [Castillo & Hartson, 2006] We focused on a user-reported critical incident method because it can identify real usability problems encountered in real work settings. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. 2 Our study was not a traditional summative study with statistically-significant results, but an exploratory study with the goal of revealing insight and understanding. The good news from the study is an indication that users with no background in usability engineering or human-computer interaction, and with the barest minimum of training in critical incident identification, can identify, report, and rate the severity level of their own critical incidents. The success of the user-reported critical incident method depends on this important indication. In particular, users were able to identify most of the medium-and high-severity critical incidents, and most of the problems not reported were of low severity. Users took almost twice as long as we expected (an average of 5 ½ minutes compared to our expected 3 minutes) to make a critical incident report, but few users reported that reporting was intrusive or interfered with work tasks. All of our users reported a satisfaction in being able to report problems to evaluators and designers. In the second phase we looked at whether evaluators can use remotely reported critical incident data to produce usability problems descriptions. Although data in …